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Background 


Since 2011, the Smithsonian Center for Learning and Digital Access (GCLDA) has strived to 
better understand and address the needs of educators utilizing digital assets through a variety 
of research and user testing studies that have led to the creation of a new digital learning 
platform, the Smithsonian Learning Lab (SLL)'. The Smithsonian Learning Lab provides access 
to the digital resources from across the Smithsonian’s 19 museums, 9 major research centers, 
and the National Zoo, to be used as real-world learning experiences. With a repository of over 
1.6 million objects and a new resource being digitized and added every 6 seconds, the 
Learning Lab provides specialized tools to aid in the discovery and creative use of its rich 
digital materials. For students using the Learning Lab, it is designed to aid in building lasting 
knowledge and critical skills that take learners from simply finding resources to thoughtful 
selection, examination, organization, and creation of new resources. 


The SLL, as it currently exists, was largely informed by the input and practice of diverse and 
effective educators.” Therefore the goal of these Student Testing experiences, as a piece of a 
larger research effort, Understanding the Needs of Student Users of Digital Smithsonian 
Resources, focuses on testing possible approaches to using the SLL in the classroom by 
students (based on the other components of this project: vols 1-5). While not intended to be a 
mere validation of SLL’s features, the design for this review and summary report is to lend 
additional insight into how digital systems, tools, pedagogy and content, can be adapted to 
better meet students' learning needs. As educational psychologist Paul A. Kirschner. points 
out, “If the student is viewed as the end user... participatory design needs to include a more 
direct participation/contribution of the student in the design of (technology enhanced) learning 
environments”. The overall project will assimilate the findings of Student Tests and 
Observations along with other research to address some key questions around methods or 
requirements for enhancing student motivation and engagement with digital content and tools. 


1. What are the ways that students engage with digital content in academic settings? 

2. What are the motivations for student use of digital content? 

3. What are the interface requirements/scaffolds needed to enable and enhance student 
engagement with rich digital resources? 


' Milligan, D., and M. Wadman, M. 2015. "From Physical to Digital: Recent Research into the Discovery, 
Analysis. . .” 
<http://mw2015.museumsandtheweb.com/paper/from-physical-to-digital-recent-research-into-the-disc 


overy-analysis-and-use-of-museums-resources-by-classroom-educators-and-students/> 
2 "smithsonian-digital-learning - Teacher Toolkit (Research Findings)." 2013. 28 Sep. 2016 


<https://smithsonian-digital-learning.wikispaces.com/Teacher+Toolkit+(Research+Findings)> 
3 Kirschner, P. A. 2015. "Do We Need Teachers as Designers of Technology-Enhanced Learning?” 


<http://link.springer.com/article/10.1007/s11251 -015-9346-9> 


Testing Process Description 


Assessing information from the Environmental Scan (Env. Scan), the Literature Review (Lit. 
Review), and Observations of Pilot in Pittsburgh, PA Classrooms (Pitt. Obsv.), targeted features 
were selected for further classroom prototyping and testing with students ranging from ages 
13-17 years old in 8 separate classroom sessions at 3 diverse school sites. Below is 
documentation on the elements taken into consideration for testing and the various 
approaches that were used in each classroom. 


escription of Classes Observed 


, Group 1 : Group 3 

Class A & B: 10th Grade - World Literature Class A & B: 11th Grade - American Literature 
(Comprehensive High School / 2 Classes) (Charter High School / 2 Classes) 

, Learning Collection: Lord of the Flies - Relation to : Learning Collection: Transcendentalist Thinking 

: Totalitarianism _ http://learninglab.si.edu/a/Il-c/A2doj3Jgb7xYYhPV 


http://learninglab.si.edu/q/ll-c/8YUt6uULGPhj16CYk 


: Group 2 | Group 4 

| Class A & B: 10th Grade - World History | Class A & B: 8th Grade - US History/Science 

| (Charter High School / 2 Classes) _ (Charter Middle School / 2 Classes) 

| Learning Collection: _ Learning Collection: 

Rise of Industrialism My Race to Space Collection - Space Race of 60’s 
| http://learninglab.si.edu/q/ll-c/KiPRhtKWnMn10101 | http://learninglab.si.edu/q/II-c/JVbfHRJ4WdUkf1kJ 


Main Elements Considered for Testing 


Items 1-10 listed in the table below represent elements identified for potential testing as 
identified from the Literature Review (Vol, 1), the Environmental Scans (Vols. 2-3, and the 
Observations of Pilot in Pittsburgh, PA Classrooms (Vol. 5). We selected elements for testing 
that were largely dependent on the student's’ interaction with the platform and content. 


Conversely, those elements determined to be already under development or exclusively related 
to inputs of the teacher were avoided. We selected elements that could be adequately tested 
and observed in a classroom setting. 


Target Element / Lit. = Already Largely Student Can Be 

Feature / Tool Review Being Dependent Controlled / | Readily 
Developed on Teacher/ | Focused Prototyped 

Designer Element 

1. Learner ability to 

record information as 

notes / Share notes 

2. Learner ability to X X 

monitor progress / 

expose progress to 

others 


3. Students 
communicate and 
share work product 


4. User provided 
modeling on how to 
best engage 
community 


7. Students can work 
collaboratively on 
projects or 
assignments 


8. Prefer access to 
print-based 
worksheets or digital 
version integrated 
in-system 


9. Share resources to 
others outside of 
system 


10. Core experience 
can be embedded in 
other environment / 

platform 
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Potential Test Elements Design Approach 


The following table shows elements identified for testing, as referenced above numerically. In 
order to create as much flexibility as possible for diverse classroom testing environments, dual 
approaches were designed for each item to be examined. Texts in BOLD represent testing 
performed and approach. 


Approach 1 (Lo-Fi) Approach 2 (Hi-Fi) 


Test Elements 


1a. Document notes, 
findings, etc. for 
self-regulated reflection 


1b. Document notes, 
findings, etc. for sharing 
with others (instructor, 
peers, community) 


2a. Progress Indicator 
available to student for 
self-monitoring 


1. Notepad (paper / pencil) on desk 

2. Sticky notepad (stick on right or left 
side of screen) 

3. Post-It notes (organized on desks 
separate from device in some 
relational order) 


At end of tasks, ask students to take 
option 2. and 3. Sticky notepad from 
above and have students select those 
they think would be interesting, useful, 
or represent their best thinking for 
audience of instructor, and then 
audience of peers and stick on board 
by question. (Use board in class for 
students to submit their responses) 


-If we want progress to be more privately 
documented, we can employ small sticky 
dots and have students move them from 


1. NotePad or TextEdit, digital word 
processor/text editor functionality, open on 
device oriented to where they want it to 
“stay” throughout work. Or 

Google Doc to record notes or question 
responses. Could give them choice of 
minimize and toggle, or resize browser 
and make persistently visible either as 
side by side with learning 
resources/content or stacked, and show 
both and then observe and record what 
students end up selecting. 


2. We could add as open-ended question 
type of various resources and title as 

Notepad or Notes in addition to persistent 
Notepad app and see which they prefer to 


generate content into or just make it an 
additional option. (Key is that notes taken 
on a single resource, don’t carry to other 
resources while moving forward) 


Since Hi-Fi options above conclude with 
students only having in-system or 
on-device digital documentation, we could 
have them categorize as Share to Teachers 
or Share to Peers with similar framing as 
that to the left of this and submitting via 
email. 


If we have classes in which students create 
Learning Lab accounts ahead of time, then 
we can have them respond to questions 
within the system. 


-simple “been here” icon or shading for 
resources visited already. 
-simple “been here” icon or shading for 


2b. Progress indicator 
available to community for 
sharing 


3. User Sharing of findings 
or resulting work with 
others 


7. Users collaborating on 
work or performing tasks 
together 


one side of the laptop keyboard to the 
other as they complete each section of 
the task. 


-or we can have them pull a small but 
visible sticker from a sheet and place 
on top of their screen. 


Tested this in unison with public 
sharing of progress using method in 
2b. 


- Clear small cups on desk with skittles 
(2-3) in each correlating to each step in 
task. Students eat contents in sequential 
order probably in relation to tasks 
completed. 

This is visible to observers and others. 
(could incite competition, and maybe 
then become an alternative factor we 
aren’t trying to look at necessarily, but i 
guess that is again, something we can 
determine is an outcome of visible 
indicators for below. 


- Sticky tabs across tops of monitors 
for each segment of the learning 
activity (collection) the students are 
completing...students move a sticky 
tab over to opposite side of their 
monitor as they complete each 
segment. 


-Students in teams of two and can share 
their work verbally or through shared 
writing pad. 


-If findings are simple results, perhaps 
we can have students write findings 
on post-it notes and place in large 
area on board as they move through 
exercises. 


If students’ deliverables include just 
formative and summative questions, the 
student teams can collaboratively 
answer. 


If students deliverables include 
development of a product in the form of 
a presentation, or a collection of their 
own, one student will need to create 
while other contributes. 


questions or info. Or hotspot tabs added to 
resource already visited by student. 
-simple progress bar that is persistent 
(perhaps at bottom), and merely renders as 
scaled incremental dynamically in relation 
to # of resources in collection and shows 
progress for # of resources already visited 
-complex progress monitoring that more 
granularly shows progress through 
percentage of questions answered. 


Won't be able to integrate this into digital 
environment for testing nor have 
comparable utility to achieve for whole 
class, in-system viewing. 


-Teacher “turns on” student response 
sharing on questions. Peers’ responses 
can be shown to students after they 
submit their own response as an option. 


- Group discussion type response can be 
activated where all students contribute 
findings as part of a group dialogue 


If students deliverables include just 
formative and summative questions, the 
student teams can collaboratively 
answer in a Google doc. 


Can invite students to work in pairs or 
remain alone. Can observe where and 
how students engage in work 
cooperatively, or how they divide 
tasks, etc... 


8. Student use of paper Provide sticky Notepads for students No Hi-Fi options available, so method were 
materials over digital to record information on in addition to | as follows: 


materials online form. e Notepads will be stuck on each 

desk and students will be told 

Provide questions as paper handouts that they may use them to jot 

with room to answer questions in down any information they need 

addition to digital means to review to during the tasks. 

and respond to questions Handout version of 
assignments/tasks will be made 
available for those students 
wanting one by way of stack at 
front of class. 
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Synthesized Classroom Observation Notes 


Group 1 Choose to 


Class A Work Together 


23 Students 


es 
es 


Engagement 
Level 


Having the worksheet on my 
laptop was easier to use 

| liked being able to put 
answers directly next to the 
video we were watching 

Like the video, but not enough 
time, so need to watch again 
Having the option of paper 
notes and online notes is good 


Not sure why google doc 
was separate from site 
Not enough time to finish 
all the questions 

Hard to get my account 
started 

My computer kept going 
off the internet 


Group 1 Work Together 
Class B 
29 Students 


rasnc 
ener fa 


Site / Students ftom 


Group 2 
Class A Together 
8 Students 
= Notes 


|Online Notes Notes 


Engagement 


| like having online videos and 
questions instead of just 
worksheets 

Cool to help out on a project 
for all the other kids in the US - 
System was cool 

Like being able to work with a 
partner on the questions 

Wish we had more time to look 
up more stuff from all the 
Smithsonian museums 


The 3d scans sound really cool 
and | look forward to seeing 
those 

This experience really helped 
me with my note taking skills. It 
was way effective for me to 
look at the images and take 
notes of what | saw 

Taking notes on doc while 
teacher is talking, seeing others 
responses while writing your 
own 

There was nothing really bad it 
was pretty self explanatory and 
easy to understand 


| wish we had headphones 
for the video, hard to hear 
with everyone else 
watching too 

Need more time to finish 
questions 

We should set up 
accounts like day before 
we start this 


Not a big problem - as far 
as getting to the set and 
making it your own it was 
slightly complicated 
Difficult to find my 
collection again 
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| loved how easy it was to take 
notes while looking at the slide. 
It was easy to use! 

| liked how we could see the 
image on our own computers 
while we talked about it. 

The basic design and home 
page was very nice. The search 
engine was nice 

| really like the whole idea of 
being able to take notes online 
using technology and you guys 
are easy to understand 


e 


Group 2 Worked 
Class B Together 
7 Students 


Paper Notes 


Online Notes 


Engagement 
Level - 
Med/High 


Group 3 Worked 
Class A Together 
16 Students 


Paper Notes 


4 
3 


It was really easy to take notes 
online and hearing the teacher 
speak. As a visual and audi 
learner | like it. 

| liked how easy the note-taking 
was. It was easy to understand 
| like that | can have the 
powerpoint pictures and note 
outline right in front of me 
during the lecture. 

| loved the idea in theory as 
well as the clean and neat 
webpage overall. | did enjoy 
using pictures instead of text as 
well 

Good color scheme 

A charming home page that 
covers the three things (create, 
share, discover) that we expect 
to see. | really like the copying 
ability to be able to edit. The 
search engine is very precise. 
There is a large variety. 

Site is well optimised. 


| found it very easy to use and 
liked having all the resources 
right there in one place 

The whole site (the format) was 
easily navigated to me. | had no 
problems. 


| felt it was hard to select 
and label the boxes. As 
well as this | am not sure if 
my notes saved. 

Is there a next slide 
option? 

Unclear for viability with 
students using website to 
take notes. Are the copied 
collections synched with 
the teachers? This could 
be useful for better 
communication with the 
teacher and student. | like 
the indifference between 
teachers and accounts. 
We may need slightly 
more thorough spell 
checker. 


When | moved from the 
video to the cartoon, it 
didn’t save my answers to 
the video. 

Despised how the 
quotations system 
worked. 

Add a search bar to 


Ae 


Group 3 
Class B 
9 Students 


Engagement 
Level - 
Med/High 


Worked 
Together 


Engagement 
Level - High 


Liked the simplicity of the lab. 
| liked how | was able to pull up 
different things within the 
website easily. 

| like how the youtube videos 
would be played on the site 
and resized when a side note 
was opened. 

Heart the teacher’s notes on 
pictures.|t is very easy to use 
Can easily navigate site. 


Easy to navigate 

Having my notes and answers 
saved with our stuff helps 

| like how the transcendental 
quotes were connected to 
media 

Online material is more 
engaging for me at least 
Liked working together on the 
assignment 


search related 
topics/quotes. 
e Add transcripts for videos. 


Video wouldn’t play on my 
laptop, but did for my 
partner 

Didn’t know if my answers 
were submitted to my 
teacher or not 

Searching brought up a lot 
of stuff unrelated to what | 
was looking for 


Group 4 
Class A 
25 Students 


Create an 
Account 


Copy a 24 
Collection 
Add Resources 


Collection with 
Someone 


| like that the site is easy to use 
and has lots of things on it 

| like all the drawing that | can 
find. 

You can see a lot of things that 
you wouldn’t think were ina 
museum. 

| liked how this website was so 
easy to use. 

Good you can personalize your 
collection. 


liked looking at pictures of space. 


| liked the simplicity. 

| liked being able to make a 
collection of stuff that | liked 
about space. 


It should ask if you 
mis-spelled something like 
ask if this is what you 
want. 

If you searched something 
and spelled it wrong it 
wouldn’t come up with 
something wrong 

One thing | did not like 
was how there was no 
autocorrect when 
something was spelled 
wrong. 

If you spell something 
wrong it won’t come up. 

| didn’t like making my 
own collections. It was 
really confusing. 

| didn't like that you 
couldn’t see the image 
before you added it to the 
collection. 
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| did not like not being 
able to look at it. 


Group 4 Create an 
Class B Account 
28 Students 


| liked how easy it was to use. 

| like how easy it is to search 
for different things. 

| liked how it was easy to find 
pictures. 

| loved all of the information on 
the site really cool. 

| liked that we got to 
experience this site. 

| like the website format. 

It was all pretty good. Thanks! 
Liked cool information and 
images. 

| liked how you can publish 
your collection to help other 
people. 

| like when you search 
something it gives all the most 
possible answers and shows a 
lot. 

| like how you are able to make 
your own collections and 
change your profile picture. 


There was no spellcheck. 
This would be very, very 
helpful. 

Didn’t like that it didn’t 
always show right results. 
| think you should be able 
to just click anywhere on 
an image to drag it. 

Mine took long to load. 
But | was almost done 
anyway. 

| think perhaps your 
“share” symbols should be 
changed. It was hard to 
find. 

It needed to be more 
direct to find your 
collection. 

| didn’t like how when | 
share it via email it took 
me all the way to gmail. 
It’s kind of confusing to 
search because so many 
things show up. 

| didn’t like how once | 
searched for pictures 
outside of the earth and it 
showed me random 
people. 
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Observed Trends 


The following list represents a summary of conclusions found as a result of the synthesis of this 


data. 


Student-Use Trends Supported by Observations: 


Test Items 


1a. Document notes, findings, etc. 
for self-regulated reflection 


1b. Document notes, findings, 
etc. for sharing with others 
(instructor, peers, community) 


2a. Progress Indicator available to 
student for self-monitoring 


2b. Progress indicator available to 
community for sharing 


3. User Sharing of findings or 
resulting work with others 


7. Users collaborating on work or 
performing tasks together 


8. Student use of paper materials 
over digital materials 


Resulting Observations 


+ Majority of students made use of digital note-taking tools when available. 


- Majority of students DID NOT take advantage of reviewing peers responses 
or contributions when provided access. 


* Students use of progress monitoring for self-tracking and/or as visual 
indicator to others was not effectively tested, nor was the duration of the tasks 
sufficient to model use-cases found in Literature Review in which students 
were engaged in projects that spanned multiple class periods up to multiple 
weeks of sustained coursework. 


* Students use of progress monitoring for self-tracking and/or as visual 
indicator to others was not effectively tested, nor was the duration of the tasks 
sufficient to model use-cases found in Literature Review in which students 
were engaged in projects that spanned multiple class periods up to multiple 
weeks of sustained coursework. 


+ Majority of students were able to set up personal accounts to record 
information and submit to teacher when made available. 


- Majority of students DID NOT opt to work collaboratively when allowed. 


+ Majority of students preferred in-system questions and guidance instead of 
paper-based worksheet when both were available. 
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